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Abstract 


In 2015, 26 two- and four-year institutions received grants to help implement and 
sustain reforms consistent with the Integrated Planning and Advising for Student Success 
(iPASS) reform approach. Beginning in 2015, the colleges worked to launch or enhance 
technologies and related structural and procedural changes that would enable them to 
provide holistic, long-term support to all students by 2018. CCRC is analyzing key 
performance indicators (KPIs) of short- and long-term student outcomes at these 
institutions—including those measuring credits earned, GPA, progress in developmental 
and gateway courses, retention, and completion—to better understand progress made 
under the advising reforms. 

Recognizing an institution’s baseline level of performance prior to implementing 
a reform is critical to determining the reform’s effectiveness. This paper provides 
baseline KPIs for 22 of the 26 colleges that were awarded an iPASS grant. We examine 
trends in institutional outcomes in the years prior to the start of the iPASS grant period. 
Our data indicate that, prior to the start of their funded iPASS reforms, grantee colleges 
exhibited wide variation in KPIs across institutions. We also find that four-year 
institutions generally exhibited higher performance than two-year institutions. Finally, 
our analysis of multiple KPIs across a time period prior to the iPASS grant period 
establishes that outcomes on these measures remained relatively stable for several years 
across the institutions. This stability will allow us to better interpret changes in the KPIs 


that may occur after the reforms are fully implemented. 
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1. Introduction 


Spurred by a shift in focus from college access to credential completion, colleges 
are looking to redesign advising and student support services in order to keep students on 
track to graduation. Many of these efforts incorporate technology to expand institutions’ 
capacity to provide all students with long-term, intensive, and personalized support. The 
approach known as Integrated Planning and Advising for Student Success (iPASS) 
represents a comprehensive strategy for reform that combines technology, structural 
change, and new forms of student—advisor engagement with the goal of increasing 
student persistence and success in college (Kalamkarian, Karp, & Ganga, 2017; Karp, 
Kalamkarian, Klempin, & Fletcher, 2016). While some components of the iPASS 
approach appear promising (Fletcher, Grant, Ramos, & Karp, 2016), an evaluation of a 
comprehensive technology-mediated advising redesign has not yet been undertaken. 

To help launch, establish, and study the utility of technology-mediated advising 
redesigns, the Bill & Melinda Gates Foundation and the Leona M. and Harry B. 
Helmsley Charitable Trust provided grants to 26 two- and four-year institutions to help 
implement and sustain iPASS reforms. Beginning in 2015, grantees worked to launch or 
enhance technologies and related structural and procedural changes that would enable 
them to provide holistic, long-term support to all students by 2018. As part of the grant, 
colleges were provided with technical assistance from two organizations, EDUCAUSE 
and Achieving the Dream. They were also expected to participate in a range of evaluation 
activities conducted by the Community College Research Center (CCRC) at Teachers 
College, Columbia University. ! 

CCRC’s activities include analyzing key performance indicators (KPIs) to 
determine if the 26 institutions are making progress with their advising reforms. We 
examine selected short- and long-term indicators before and after a reform is 
implemented to provide evidence of changed student outcomes. The reason for using 
both kinds of indicators is straightforward. Relying on long-term measures alone would 


mean that practitioners, policymakers, and researchers might have to wait years in order 


'In addition to the analyses presented here, some colleges participated in in-depth qualitative research. 
Others participated in a randomized control trial aimed at estimating causal impacts conducted in 
partnership with MDRC. A description of the study can be found at https://ccrc.tc.columbia.edu/research- 
project/integrated-planning-and-advising-services. html. 


to determine if a reform is working. Measuring near-term indicators can give some 
provisional indication about whether reforms are working sooner, and they can also be a 
useful tool to continuously improve new programs and policies (Jenkins & Bailey, 2017). 
Understanding an institution’s baseline level of performance prior to 
implementing a reform is critical to determining the reform’s effectiveness. This paper 
provides baseline KPIs for 22 of the 26 colleges that were awarded an iPASS grant. We 
examine trends in institutional outcomes prior to the start of the iPASS grant period in 
2015. Our data indicate that, prior to iPASS-funded reforms, grantees exhibited wide 
variation in indicators of interest across all institutions. We also find that, as is the case 
nationally, four-year institutions generally exhibited higher performance than two-year 
institutions. Finally, our analysis of multiple KPIs across time in advance of the iPASS 
grant period establishes that the outcomes on these measures remained relatively stable 
for three years across the institutions prior to the grants. This stability will allow us to 


better understand changes that take place after the reform is fully implemented. 


2. The Approach: Redesign of Technology-Mediated Advising 


Ample research indicates that advising and student support are not optimized at 
most institutions, and that this may contribute to low levels of student persistence and 
completion (Bailey, Jaggars, & Jenkins, 2015; Grubb, 2006; Jaggars & Fletcher, 2014). 
Technology-mediated advising leverages efficiency created by technology to facilitate 
the type of support that research suggests leads to improved student outcomes. The goal 
of the iPASS approach is to improve students’ advising experience by shifting the 
emphasis in advising from registration and enrollment functions toward the development 
of sustained, strategic, integrated, proactive, and personalized (SSIPP) relationships 
between students and college personnel (Kalamkarian, Karp, & Ganga, 2017; Karp & 
Stacey, 2013). This enables advising relationships to focus less on administrative tasks 
and more on developing students’ reflective and metacognitive skills, a formulation often 
referred to as advising-as-teaching (Appleby, 2008). 

SSIPP advising requires a comprehensive shift in the organization and delivery of 


student support. It cannot be fully realized without the implementation of new policies 


and procedures to guide advising that may include a case management approach or 
mandatory advising meetings. SSIPP advising also requires advisors to develop new 
skills and methods, such as regularly reaching out to students, engaging in conversations 
about academic and nonacademic challenges to completion, and entering case notes into 
advising software. 

These shifts can be encouraged and supported by technology tools that streamline 
processes, create new information channels, and perform administrative tasks more 
efficiently. Ideally, technology can reduce advisors’ administrative workloads and give 
them time, space, and resources to help more students choose majors and careers, find 
support in times of need, and graduate in a timely manner with a plan for the future. As of 
2014, there were over 100 advising-related technology tools, many with similar 
functions, made available mostly by vendors and put in use in U.S. postsecondary 
institutions, a number that has only grown since then (Tyton Partners, 2014). 

Advising and student services technologies generally fall into one of four 
categories, and the colleges examined here had been using various combinations of these 
at the time they joined the iPASS project. Early alert tools and predictive analytics allow 
advisors, faculty, and staff to identify at-risk students. They enable individuals and 
institutions to intervene before a student falls off-track and to target resources so that 
students most in need of assistance receive help. Degree planning tools enable students to 
engage in long-term program planning that aligns with institutional graduation 
requirements. By enabling students to independently schedule courses early in their 
educational careers, these tools help advising sessions move away from immediate course 
registration and toward discussion of academic progress, career planning, and problem 
solving. Finally, “case management” (or communication) platforms enable students, 
faculty, staff, and advisors to communicate across offices and services. They enable 
shared information about students, their needs, and their use of services over time. 

The presence of these technologies can serve as a catalyst for colleges to 
fundamentally redesign their advising and support services to move toward a much more 
intensive and personalized case management model (Karp et al., 2016). Note, however, 
that a technology launch is not the end goal of iPASS reform, nor is it enough on its own 


to ensure fundamental changes in advising and student support practices. Rather, by 


leveraging technology tools, advising and student service programming can undertake 
changes that transform the experience of students. Previous research (Fletcher et al., 
2016; Karp et al., 2016) has found that these changes are ideally multidimensional and 
integrated, encompassing structural change, including the use of new policies, as well as 
behavioral change, such as student support staff conversing with students about non- 
academic issues that impede their persistence. For example, using an early alert system to 
email students who did not complete a degree plan is likely to be less impactful than 
using the early alert system to facilitate personalized advising for those students to create 


long-term plans and to monitor their completion. 


3. Grantees’ Pre-Grant Use of Technology-Mediated Advising Reform 


In 2015, the Bill & Melinda Gates Foundation and the Leona M. and Harry B. 
Helmsley Charitable Trust provided 26 institutions with iPASS grants to allow them to 
implement new technologies or enhance existing ones. Grantees used the funding to 
support college-developed projects addressing education planning, advising, counseling, 
coaching, risk targeting, and intervention. Importantly, the vast majority of colleges in 
our sample had already begun some form of technology-mediated advising reform prior 
to the start of the grant period. In fact, some of the colleges in this study had previously 
received a Gates Foundation grant to implement advising technology. 

We conducted a document review of project proposals and funder updates in part 
to identify whether the colleges were using the iPASS grant to implement new 
technologies or to enhance or replace pre-existing ones. We found that only four of the 26 
grant recipients had never implemented any advising technology, so we chose to 
categorize the colleges by their prior experience with technology-mediated advising 
reform. We found that most colleges were already using more than one technology to 
improve advising (see Table 1). Using a rubric to score colleges’ experience and 
implementation levels, we grouped the colleges into four “new implementation” colleges, 


15 “launch” colleges, and seven “early adopter” colleges. 


Table 1 
iPASS Colleges’ Experience With Advising Technologies Prior to the Start of the iPASS 
Grant Period 


Number of 

Type of College Definition Colleges 
New Institution was not using any advising technologies prior to receiving the iPASS 4 
implementation grant in 2015 and plans to implement them for the first time during the grant 

period. 
Launch Institution had one or more advising technologies that were up and running 15 

prior to 2015. These colleges purchased and rolled out advising technologies on 

campus, but advisors were not actively using them with students. These colleges 

are beginning to implement the technology during the grant period. 
Early adopter Institution had one or more advising technologies that were up and running 7 


prior to 2015. In addition, end-user staff and students utilized at least one of 
these technologies at scale or for a majority of students. These colleges are 
replacing old technologies or adding additional technologies during the grant 
period. 


In future reports we will describe changes in implementation and outcomes by the 
groups listed in Table 1. We anticipate that institutions that were early adopters will be 
more advanced in iPASS implementation because they got a head start. Likewise, if these 
colleges are implementing the program better, we anticipate that these early adopter 
colleges may also be more likely to improve outcomes than colleges in the other 
categories. 

Most colleges were implementing multiple technologies prior to receiving the 
iPASS grant. We divided the advising technologies into the following categories: early 
alert tools, predictive analytics, degree planning tools, case management platforms. For 
this baseline report, we counted the number of technologies that were implemented prior 
to colleges’ receipt of their iPASS grant through our document review of project 


proposals and funder updates. 


4. Study Design 


To document changes in student outcomes as a result of technology-mediated 
advising reforms at iPASS grantee colleges, CCRC is tracking KPIs over the course of 
the grant period. Using KPIs to measure expected outcomes necessitates identifying 
performance indicators that we can reasonably expect to be influenced by the reforms. 
Figure 1 shows a generalized logic model for the types of inputs, activities, and outcomes 
expected from technology-mediated advising.” 

Building off this model, with a focus on outcomes, we created a table that aligns 
specific components of the iPASS model with KPIs that they theoretically affect. A list of 
all of the KPIs that are tracked through this project is provided in Table 2. Ideally, all of 


these should increase over time following implementation of iPASS reforms. 


Figure 1 
Logic Model of Technology-Mediated Advising Interventions 


Inputs Activities Outputs Outcomes 


Holistic, long- 


term support Improved 


student 
outcomes 


to all 
students 


> An explanation of this logic model design and its use can be found in Lawton, Brandon, Cicchinelli, and 
Kekahio (2015). 


Table 2 
Tracked KPls and What They Measure 


KPI Categories What Each KPI Measures 
Credit momentum % of students who attempted 15 or more credits during the first term 
Credits earned Average number of credits earned in Year 1 


% of credits attempted that were earned 
Grade point average (GPA) Average first-term GPA 
Average cumulative GPA 
Dev. ed. progression in English and math % of students not college ready 
% of students who attempted English or math dev. ed. in Year 1 
% of students who completed dev. ed. In Year 1 
Gateway course progression % of students who required gateway courses in Year 1 
% of students who attempted gateway courses in Year 1 
% of students who completed gateway courses in Year 1 
Retention % of students retained after Year 1 
Completion % who completed associate degree in 100% time 
% who completed associate degree in 150% time 


% who completed bachelor’s degree in 150% time 


Note: These KPIs also align with indicators of momentum (Bailey & Jenkins, 2017) and the Bill & Melinda Gates Foundation’s 
momentum metrics (Janice & Voight, 2016). 


Over the course of the study we will release two reports that track changes in 
KPIs from their baseline state (pre-grant) prior to the implementation of iPASS to a time 
when iPASS has been fully implemented. This change-over-time framework enables us 
to identify shifts in institutional performance as advising reforms take root and are 
refined. 

KPIs are reported at the institutional level. Thus, in order to observe the desired 
changes, the iPASS reforms must not only be implemented by each college, but they 
must also be implemented for most students. Implementation for most students, which we 
refer to as implementation at scale, means that the technology tools a college uses are 
deployed and used in a way that makes a meaningful difference in the student experience 
and are used by most students, faculty, and staff. 

Although KPIs are useful for identifying trends in institutional performance, they 
are imperfect measures of an intervention’s true effect. Trend lines may change for many 
reasons, including the influence of other reforms, changes in enrollment patterns, or 
external policy conditions. Moreover, given the diversity among colleges in terms of their 


size, sector, students served, and the range of advising reforms implemented, we are 


unable to provide a direct assessment of each reform’s impact. Further, it is not possible 
to make comparisons in performance between colleges, even with common data 
definitions. The colleges were performing differently at the beginning of the study, so 
any difference in performance over time could be due to systematic differences between 


the colleges at the start of the study. 


5. Study Data and Student Sample 


This report focuses on a descriptive analysis of the baseline trends in KPIs for 
colleges prior to the iPASS grant period in 2015. The student data we received concern 
metrics that have been standardized according to definitions used across the Bill & 
Melinda Gates Foundation’s Postsecondary Success portfolio.’ Institutions worked with a 
third party to create the necessary datasets and upload them to a secure website for 
validity checks. Once the data quality was assessed, the third party then transmitted the 
data to CCRC. 

We supplemented this student-level data with information about institutional 
characteristics from the Integrated Postsecondary Education Data System (IPEDS) as 
well as with information about institutions’ technology choices based on a document 
review of project proposals and funder updates to categorize the status of the 
implementation of advising reforms before the start of the grant period. 

Our sample for the study included only first-time-in-college (FTIC) students from 
each academic cohort between 2011 and 2014. Students were divided into cohorts based 
on the academic year in which they entered the college. We aggregated the student-level 
data by cohort and institution in order to create cohort average metrics at each college in 
the years prior to the start of the iPASS grant period. We limited the sample to exclude 
students who had transferred in or had previous (non-dual enrollment) college 
experience. Institutions also excluded all students who were actively dually enrolled at 


high schools from the data collection, but they included current college students who had 


3 For more information, visit https://www.gatesfoundation.org/What-We-Do/US-Program/Postsecondary- 
Success. 

‘For example, the 2011-12 cohort refers to students who began college for the first time ever during the 
summer 2011, fall 2011, or spring 2012 semesters. 


dually enrolled in the past. Only after all identifying information was removed and data 
were cleaned were they shared with CCRC. The final dataset covers the spring semester 
of 2015. 

In Table 3 we display students’ background characteristics for the entire sample 
of 22 institutions for which we received student data between 2011 and 2014.° We 
divided students’ characteristics into measures of enrollment intensity, residency, age, 
gender, race/ethnicity, socioeconomic status (SES), and academic preparation. We 
divided the sample by two-and four-year colleges because each sector tends to serve 
different student populations. 

Table 3 shows that the majority of students in two-year colleges were part-time 
students (80 percent), while the majority of four-year college students were attending 
full-time (82 percent). Two-year colleges served a higher share (94 percent) of in-state 
students compared with four-year institutions (82 percent). Students in four-year colleges 
tended to be younger than those in two-year colleges. Ninety percent of students in four- 
year colleges were 19 and under compared with 62 percent of students in two-year 
schools. Seventeen percent of students in two-year colleges were 20—24 years old, and 20 
percent of students were 25 years old or older. Students of color made up the majority of 
students at both two-year and four-year colleges in the sample. The two-year colleges 
tended to have a larger percentage of Black students, while the four-year colleges tended 
to have a larger percentage of Asian students, Hispanic students, and students of mixed 
race/ethnicity. About 33 percent of the two-year and four-year students in the sample 
were the first in their family to attend college. A larger share of two-year students (26 
percent) came from neighborhoods with a household income of less than $35,000 
compared with four-year students (17 percent). 

The student body at the colleges also differed in academic preparation. Two-year 
students were less likely to have earned a high school diploma, but they were more likely 
to have dually enrolled compared with four-year students. In English and math, two-year 
students were less likely than four-year students to be college-ready at entry overall. 
Within each sector, students were more likely to be college-ready in English than in math 


at entry. 


Four of the 26 grantee colleges did not report data. 


Table 3 
Descriptive Characteristics of iPASS Colleges, by Sector 


Variable Two-Year (%) Four-Year (%) 
Enrollment intensity in first term 

Full-time (12 or more credits) 20 82 

Part-time (fewer than 12 credits) 80 18 
Residency 

In-state 94 82 

Out-of-state 3 14 
Age 

19 and under 62 90 

20-24 17 

25 and older 20 6 
Gender 

Female 51 53 

Male 47 47 
Ethnicity 

American Indian 3 3 

Asian 6 10 

Black 22 10 

Hispanic 13 15 

Mixed race/ethnicity 4 9 

Native Hawaiian 1 0 

Non-resident alien 2 4 

White 43 42 

Missing 7 8 


SES indicators 
First-generation 


First-generation student 34 32 
Not first-generation student 34 56 
Missing 32 12 


Neighborhood income 
Household income less than $35,000 26 17 


Academic preparation 
High school graduate 
High school diploma 78 91 
Missing 14 3 
Dual enrollment 
Past dual enrollment 10 4 
English readiness at entry 


College-ready 42 67 

Not college-ready 48 9 

Missing 11 24 

Math readiness at entry 

College-ready 32 60 

Not college-ready 57 16 

Missing 11 25 
Number of students 126,766 315,266 
Number of institutions 8 14 
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6. Baseline Trends in KPIs 


In this section we present our baseline results on the study’s KPIs. For each KPI 
we plot the average for each of the baseline years (201 1—2014) for all iPASS colleges. 
Because two-year and four-year colleges tend to serve different groups of students, we 
present the results separately for each sector. 

Overall, our analysis found that there was considerable variation between the 
sectors. However, there typically were only small changes from year to year in KPIs 
within each sector. We also conducted an analysis of differences between new 
implementation, launch, and early adopter colleges, but we were not able to find any 


meaningful patterns in the data by these categories. 


6.1 Credit Momentum and Credits Earned 

Previous research on college success has found that what happens in the first year 
of college is important in students’ subsequent academic achievement, aspirations, and 
involvement on campus (Feldman, 2005; Pascarella & Terenzini, 2005; Tinto, 1993). 
Figure 2 examines the percentage of students with credit momentum, defined here as 
those students who attempted 15 or more credits in their first term. Students at four-year 
colleges are much more likely to attempt 15 or more credits in the first term on average 
compared with students at two-year institutions (57 percent versus 5 percent across all 
years). These differences between two-year and four-year colleges are to be expected, 
given the large differences in the percentage of full- and part-time students enrolled. 
Overall, a slightly larger percentage of four-year students achieved credit momentum 
over time. 

The average number of credits students attempted and earned in students’ first 
year across cohorts is displayed in Figure 3. The graphs show that students in two-year 
colleges earned substantially fewer credits on average than students in four-year colleges 


in the first year. The number of credits earned stayed stable over time. 
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Figure 2 
Percentage of Students With Credit Momentum, by Sector and Cohort 
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Note. Credit momentum is defined as attempting 15 or more credits during the first term. 


Figure 3 
Average Number of Credits Earned in Year 1, by College and Cohort 
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The percentage of credits attempted that were earned is displayed in Figure 4. At 
most four-year colleges, students earned about 84 percent of credits attempted; students 
at two-year colleges earned 53 percent of credits attempted. These trends are fairly 
consistent over time, but students at two-year colleges showed more improvement over 


time than those at four-year colleges. 


Figure 4 
Percentage of Credits Earned That Were Attempted in Year 1, by College and Cohort 
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6.3 Grade Point Average 

Each cohort’s first-term grade point average (GPA) is displayed in Figure 5. 
Students at four-year colleges had higher GPAs on average compared with students at 
two-year colleges (2.8 versus 1.9), and both groups’ GPAs increased over time. The trend 
lines of the GPAs for each cohort across the colleges increased slightly from 2011 to 
2014. 

The average cumulative GPA for every cohort is displayed for each sector in 
Figure 6. Two-year students had a GPA close to 2.1, and their GPAs improved slightly 
over time. Four-year students had a 2.8 GPA on average, a figure that remained stable 


over time. 
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Figure 5 


Average First-Term GPA, by Sector and Cohort 
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Figure 6 
Average Cumulative GPA, by Sector and Cohort 
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6.4 Developmental Education Progression in English and Math 

In this section we describe our findings for developmental English and math 
course taking among first-year students. Similar to first-semester student experiences, we 
see mostly flat trends for both sectors. 

Figure 7 presents six graphs describing the students’ progression in English 
developmental courses. The top row is two-year colleges and the bottom row is four-year 
colleges. The pipeline begins with the first graph in each row illustrating the percentage 
of first-year students who were not college-ready upon entry into college, as measured by 
college placement examinations. The middle graph shows the percentage of students 
deemed not college-ready who attempted developmental English. The third graph 
illustrates the percentage of students who passed developmental English among those 
students who enrolled in a developmental English course. 

The graphs show distinctly different rates of progression across the two sectors. In 
two-year colleges, a larger proportion of students were not college-ready (43 percent), 
and a similar share attempted developmental English (45 percent). A smaller proportion 
of four-year students were not college-ready (12 percent), yet the same share of four-year 
students identified as not college-ready attempted developmental English (45 percent) as 
two-year students. The completion rate for developmental English slightly increased for 
two-year college students over time (from 60 percent to 65 percent) and is constant for 
four-year college students over time but at a higher rate (87 percent). 

Similar to Figure 7, Figure 8 presents six graphs describing the students’ 
progression in developmental math courses. The graphs also show differences across the 
two sectors. In two-year colleges, a larger proportion of students were not college-ready 
in math (50 percent); however, a smaller share attempted developmental math (25 
percent). Fewer four-year students were not college-ready (15 percent), yet a greater 
share of those students attempted developmental math (45 percent). The completion rate 
of developmental math over time decreased for two-year college students (from 24 
percent to 15 percent) and decreased slightly for four-year colleges students (79 percent 


to 76 percent). 
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Figure 7 
Developmental English Education Progression, by Sector and Cohort 


Two-year Colleges Two-year Colleges Two-year Colleges 
English - Not Dev. English - Dev. English - 
college ready Attempted Completed 
100% - 100% - 100% - 
75% - 75% - 75% “60 61 63 65 
——— Oe 
50% -40 44 46 44 59%-e 44 42 42 50%- 
——————— a EE 

25% - 25% - 25% - 

0% 7 t t 1 1 0% 7 t t I 1 0% ~ 1 t t t 
© = NI fap) vt = N ap) vt = N oD wt 
re)) =) =) =) =) =) =) o o o o ry =) 
oO N NI AN NI AN N AN NI NI N NI AN 
B Four-year Colleges Four-year Colleges Four-year Colleges 
o English - Not Dev. English - Dev. English - 

o. college ready Attempted Completed 

100% - 100% - 100% -g6 89 89 86 

a en ere 
75% - 75% - 75% - 
50% - 50% 48 ——-44 S48 50% 
25% “10 9 8 8 25% - 25% - 

(ee ee 
0% : t t t ' 0% a t 1 I t 0% 7 iF t ' i 

- NWN fae) tv = N oO vt oa N oO vt 

=) o =) o =) ry to) ty ry =) ry 3S 

A al A al Al Al A Al al ial al A 

Cohort 


Note. Six colleges were not included due to missing data. “Not college-ready” is defined as students who are required to complete developmental 
English. The percentage of students who attempted developmental English is conditional on being required to take developmental English. The 
percentage of students who completed developmental English is conditional on attempting developmental English. 
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Figure 8 
Developmental Math Education Progression, by Sector and Cohort 
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Note. Six colleges were not included due to missing data. Students who are required to completed developmental math are also defined as not 
college-ready in math. The percentage of students who attempted developmental math is conditional on being required to take developmental 
math. The percentage of students who completed developmental math is conditional on attempting developmental math. 
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6.5 Gateway Course Progression 

Figure 9 and Figure 10 examine the gateway course progression—defined as the 
requirement, attempt, and completion of the first college-level course in math or English 
for any program within the first year—at each of the iPASS colleges between 2011 and 
2014. The graphs show that similar shares of students at two-year and four-year colleges 
were required to take gateway courses. ° 

Of those who were required to take gateway courses, only about 25 percent of 
two-year students attempted a course in English or math during their first year, compared 
with about 55 percent of four-year students. The percentage of students attempting a 
gateway course in English and math increased slightly over time for both sectors. In the 
two-year sector, about half of students who attempted a gateway course in English and 
about 70 percent of students who attempted a gateway course in math completed it in the 
first year. In the four-year sector, about 75 percent of students who attempted a gateway 
course in English and 88 percent of students who attempted a gateway course in math 


completed it in the first year. 


6 Some students may be exempt from gateway math coursework; for example, students transferring from 
another institution where they completed the requirement, students fulfilling the requirement by successful 
completion of AP, IB, or similar high school coursework, or those enrolled in majors that do not require a 
gateway math course. 
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Figure 9: Gateway Course Progression in English, by Sector and Cohort 
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Note. Six colleges were not included due to missing data. The percentage of students who attempted gateway English is conditional 


on being required to take gateway English. The percentage of students who completed gateway English is conditional on attempting 
gateway English. 
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Figure 10: Gateway Course Progression in Math, by College and Cohort 
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Note. Six colleges were not included due to missing data. The percentage of students who attempted gateway math is conditional on 


being required to take gateway math. The percentage of students who completed gateway math is conditional on attempting 
gateway math. 
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6.6 Retention 

We also examine the percentage of students in each cohort who were retained in 
their second year, as shown in Figure 11. On average, about 78 percent of students in 
four-year colleges were retained compared with about 43 percent of students in two-year 
colleges. The percentage of students retained was stable over time within two-year 


colleges but increased slightly at four-year colleges. 


Figure 11: Percentage of Students Retained in Their Second Year, by Sector and Cohort 
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6.7 Completion 


Two of our longer term outcomes were the percentage of students in each cohort 
that completed an associate degree in two years or less and in three years or less. We 
excluded the 2014 cohort from the analysis because that cohort had not had three full 
years to earn an associate degree when the data were collected. 

Associate degree completion. Figure 12 shows that of students in two-year 


colleges, fewer than 5 percent earned an associate degree in two years or less or in three 
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years or less. These results are lower than national comparisons. Nationally, about 12 


percent of students at two-year institutions earned an associate degree within two years 
(Ifill et al., 2016). 


Figure 12: Associate Degree Completion, by Cohort 
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Bachelor’s degree completion. Our longest term outcome, the percentage of 
students in the 2011 cohort who completed a bachelor’s in six years or less, is shown in 
Figure 13. We focus on four-year college students in the 2011 cohort because only that 
cohort originally enrolled in college six years ago as of the latest data upload. Overall, 
36.5 percent of the sample earned a bachelor’s degree within six years This result is 
lower than national comparisons. Nationally, about 60 percent of full-time students at 
four-year institutions earned a bachelor’s degree within six years (U.S. Department of 


Education, National Center for Education Statistics, 2018). 


22 


Figure 13: Bachelor’s Degree Completion Among Students in the 2011 Cohort 
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7. Discussion and Conclusion 


Tracking KPIs over time is important in understanding the year-to-year variation 
in student outcomes by sector. We see considerable differences between the two-year and 
four-year institutions under study. Four-year colleges tended to have a student body that 
was younger, more academically prepared, and more likely to attend full-time. Four-year 
students were more likely to select a major in their first term, earn more of the credits that 
they attempted, and remain enrolled in their second year compared with two-year 
students. Overall, there was little change in KPIs between the 2011 cohort and the 2014 
cohort within either sector. 

This method of tracking multiple years of baseline data can serve to strengthen 
our evaluation analyses. Studies that simply track one year pre-intervention and one year 
post-intervention do not take into account year-to-year variation. What might otherwise 
be considered a positive or negative difference could really reflect expected year-to-year 
variation. We control for this year-to-year variation by analyzing multiple years of 
baseline data. In future reports we will look for differences in KPIs from the baseline 


pattern to those patterns of future cohorts who have experienced iPASS reforms. 
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Over the next year, we will continue to collect implementation and KPI data and 
share the results with our college partners so they can adjust and adapt their reforms. We 
encourage the colleges to provide accurate and complete data for each metric; this will 
help CCRC provide more meaningful and useful results to our partners. Some metrics, 
particularly those measuring developmental education and gateway course participation 
and completion, suffer from missing data that limits their utility as outcome measures in 
future studies. The missing data are particularly problematic because they could be useful 
in the early evaluation of advising reforms. 

Additionally, we encourage institutions to explore their KPIs by looking at 
subgroups of students by race/ethnicity, gender, age, program of study, and economic 
status. Subgroup analyses can help to uncover differences in outcomes between groups 
and is thus an important consideration for faculty, staff, and administrators involved in 
reform. Future areas of CCRC research will include examining the impact of the iPASS 


grant and studying the implementation of the colleges’ reforms. 
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